The present invention relates to a monitor apparatus using an imaging unit, or in particular to an object tracking method for automatically detecting an object intruding into an imaging field or image pickup field from a video signal inputted from the imaging unit and automatically tracking the motion of the detected object and an object tracking apparatus for adjusting the imaging direction in accordance with the detected motion of the object.
A video monitor apparatus using an imaging unit or image pickup unit such as a camera has been widely used. A monitor system using a video monitor apparatus is in demand, in which an intruding object is automatically detected from the image input from an image input means such as a camera, the motion of the object is automatically tracked and a predetermined announcement or alarm action can be taken, apart from the manned monitoring system in which an intruding object such as a man or an automotive vehicle entering the monitor field is detected or tracked by a human monitor while watching the image displayed on the monitor.
For realizing such an unmanned system, the first step is to detect an intruding object in the view field by a so-called subtraction method or the like. In the subtraction method, the input image obtained by an imaging device such as a television camera (hereinafter referred to as the TV camera) is compared with a reference background image prepared in advance, i.e. an image not including the object to be detected to determine the brightness (or intensity) difference for each pixel and detect an area with a large difference value as an object. The part of the input image (hereinafter referred to as partial image) corresponding to the position of the intruding object detected in this way is registered as a template, so that a position associated with the maximum degree of coincidence with the template image is detected in the sequentially input images. This method is widely known as the template matching, and is described in detail, for example, in the book entitled xe2x80x9cIntroduction to Computer Image Processingxe2x80x9d edited by Hideyuki Tamura, published by Soken Publishing Co. pp. 149-153, 1985. Further, this method is disclosed in xe2x80x9cDigital Picture Processingxe2x80x9d published by ACADEMIC PRESS pp. 296-303, 1976 and U.S. Pat. No. 5,554,983, the disclosures of each is hereby incorporated herein by reference.
The aforementioned method of tracking an intruding object by template matching poses the problem that with the change of the orientation or posture of the target object (when the target object person turns to the right or turns around, for example), the deviation of the target object from the matching position increases to such an extent that the accurate and stable tracking becomes impossible.
Specifically, the template matching has the property that the pattern portion high in contrast in the template images coincides for matching. In the case where a vehicle is a target object, for example, substantially the whole vehicle facing forward first constituting an object of matching (the input image 802 in FIG. 8) may subsequently proceed in a different direction (orientation) and turn sideways, in which case only the front portion of the turned vehicle constitutes the object of matching. As compared with when the whole vehicle has been the object of matching, the center of matching changes to the front portion of the vehicle and therefore the displacement occurs such that the center of the detected location is deviated from the center of the vehicle to the front portion thereof.
This will be explained with reference to FIG. 8. FIG. 8 is a diagram in which a vehicle running along a curved lane within an imaging field is assumed to be an intruding object for explaining the flow of the tracking processing. Numerals 801a, 803a, 8OSa, 807a designate template images at time points t1xe2x88x921, t1, t1+1, t1+2, respectively; numerals 801, 803, 805, 807 designate images indicating the updated template images 801a, 803a, 805a, 807a, respectively; numerals 802, 804, 806, 808 designate input images at time points t1, t1+1, t1+2, t1+3, respectively; numerals 802a, 804a, 806a, 808a designate the positions of the object detected by template matching at time points t1, t1+1, t1+2, t1+3, respectively; and numerals 802b, 804b, 806b, 808b designate the positions of the template images in the immediately preceding frame (i.e. the template images at t1xe2x88x921, t1, t1+1, t1+2, respectively).
In FIG. 8, the template image 801a registered at time point t1xe2x88x921 is an image of a moving vehicle with the front portion thereof directed substantially forward. At time point t1, the template matching is conducted using this template image 801a to detect the position moved by the target object, while at the same time updating the template image 801a to the template image 803a. Then, at time point t1+1, the template image 803a is updated to the temperate image 805a. Further, at time t1+2, the template image 805a is updated to the template image 807a. By conducting this process up to time t1+3, the template matching which has thus been conducted on the front portion including the vehicles"" lights at the tracking starting time t1 is deviated to the left of the vehicle.
This phenomenon is caused by the fact that the matching is conducted in such a manner as to reduce the deviation of the positions between the input image target for template matching and the image portion high in contrast in the template image. In this case, such a portion is the light of the vehicle. As a result, in the case where the target object turns to the left in FIG. 8, for example, the deviation is leftward, and vice versa.
Further, at time t1, only the vehicle image is included in the template image 801a. With the direction change of the target object and the resulting deviation of the template position, however, the image of the background portion other than the object image occupies into the template image 807a. In the case where the tracking is continued using a template image like the template image 807a including many images other than that of the target object, the target object cannot be matched and the background portion that has occupied into the template is matched. Thus, in the case where the target object changes in direction, for example, the object tracking method using the template matching cannot guarantee the tracking of the target object and cannot assure stable tracking because the pattern of the target object apparently moves, which causes the position of the template to deviate.
Further, the aforementioned method of tracking an intruding object by the template matching, which is the process of detecting the portion of the input image associated with the maximum degree of coincidence with the template image, poses another problem that the target object, if temporarily hidden behind some blocking object, cannot be found. Still another problem is that in the case where a plurality of objects (moving objects) exist in the view field and the target object is temporarily hidden behind another moving object, the template image may be updated undesirably with the blocking object in front of the target object as a template image, thereby making it impossible to track the target object.
An example in which the target object is cannot be tracked is explained with reference to FIG. 14. FIG. 14 is another diagram for explaining the flow of the process for tracking an intruding object in the case where a plurality of moving objects are imaged in superposed relation. Numeral 1801a designates a template image of an object updated at time point t0xe2x88x921, numeral 1801 a view showing the position of the template image 1801a in the input image at time point t0xe2x88x921, numeral 1802 the input image at time point t0, numeral 1802a the position of the object detected by template matching at time point t0, numeral 1802a the position of the object detected by template matching at time point t0, numeral 1802b another moving object imaged at time point t0, numeral 1803a the template image at time point t0, numeral 1803 a view showing the position of the template image 1803a in the input image at time pint t0, numeral 1804 the input image at time point t0+1, numeral 1804a the position of the object detected by template matching at time point t0+1, numeral 1804b another moving object imaged at time point t0+1, numeral 1805a a template image of the object updated at time point t0+1, numeral 1805 a view showing the position of the template image 1805a in the input image at time point t0+1, numeral 1806 the input image at time point t0+2, numeral 1806a the position of the object detected by template matching at time point t0+2, numeral 1806b another moving object imaged at time point t0+2, numeral 1807a the template image of the object updated at time point t0+2, numeral 1807 a view showing the position of the template image 1807a in the input image at time point t0+2, numeral 1808 the input image at time point t0+3, numeral 1808a the position of the object detected by template matching at time point t0+3, and numeral 1808b another moving object imaged at time point t0+3.
FIG. 14 explains the manner in which the tracking process is conducted when a man-like object is passed by another man-like object walking from the opposite direction based on the template image 1801a of the first man-like object constituting a target object detected by the difference method at time point t1xe2x88x921. At time point t1, the matching processing is executed for the input image 1802 using the registered template image 1801a to detect the position 1802a of the target object, while at the same time updating the template image 1801a to the template image 1803a located in the view 1803. In the process, the second man-like object 1802b is distant, and therefore no problem of passing is posed in template matching. Other moving objects 1804b, 1806b passed the front of the target object at time points t1+1 and t1+2 and the position 1804a of the target object could be somehow recognized correctly in the input image 1804 at time point t1+1. In the input image 1806 at time point t1+2, however, another object 1806b is substantially superposed at the position 1806a of the target object, and therefore they cannot be distinguished from each other. Thus, as the template images 1805a and 1807a are updated successively, the image of the man-like object that has passed the front of the target object is included into the template image. Specifically, other moving objects occupy the template image in a greater proportion than the target object, until finally, at time point t1+3, the man-like object 1808b that has passed the front of the target object is erroneously recognized as the position 1808a of the target object, and this particular man-like object 1808b is tracked subsequently. The object tracking method using the template matching, therefore, cannot guarantee that a target object is tracked correctly thereby making stable tracking impossible in the case where another object passes the front of the target object.
A similar phenomenon occurs also in the absence of another moving object. The target object being hidden behind a block is a case in point. In such a case, the template image is occupied in a lesser proportion by the target object, and therefore the tracking is terminated at the particular time point or a template image is updated to a different template image than that of the target object, thereby making the tracking at the next time point difficult.
As described above, the conventional method of tracking an intruding object by template matching has the disadvantage that stable tracking is impossible in the case where the target object changes its orientation or posture considerably.
An object of the present invention is to provide a reliable object tracking method and apparatus by which the disadvantages of the prior art are obviated and an object can be accurately detected and tracked even in the case where the orientation or posture of the target object is changed considerably.
In order to achieve this object, according to one aspect of the invention, there is provided an object tracking method used with an object tracking apparatus in which a template image is registered, and an object in the imaging field is automatically detected by template matching based on the registered template image and the detected object is automatically tracked, comprising the steps of registering an image including at least a part of a detected object as a template image, detecting the position of a partial image of the input image signal associated with the maximum degree of coincidence between the registered template image and the input image signal by template matching, correcting the detected template position to the position of the object detected based on the edge image of the input image and updating the template image based on the corrected detected position thereby to track the object in the imaging field.
According to another aspect of the invention, there is provided an object tracking apparatus comprising an imaging unit for picking up an image of the monitor range for detecting and tracking an object in the imaging field, a pan and tilt head for changing the direction of the view field of the imaging unit, an image input interface for converting the video signals acquired by the imaging unit sequentially into image signals, an image processor for processing the image signals converted by the image input interface, and a pan and tilt head control interface for supplying a control signal for controlling the pan and tilt head to enable the image processor to change the direction of the view field of the imaging unit, wherein the image processor matches a template registered in advance with the image signals sequentially input from the imaging unit, the template matching position is corrected to a new template matching position associated with the maximum edge density in a predetermined range (expanded partial image) in the neighborhood of the template matching position associated with the maximum degree of coincidence of the image signal, the image at the newly corrected matching position is updated as a template, the direction of the target object is detected based on the newly corrected matching position, the direction of the view field of the imaging unit is adjusted through the pan and tilt head control interface from the direction thus obtained, and thereby the object intruding into the imaging field of the imaging unit is tracked.
As explained above, the conventional intruding object tracking method using the template matching described above also has another disadvantage that an object cannot be accurately tracked in the case where another object passes the front of a target object in relative motion.
Accordingly, another object of the invention is to provide a method and an apparatus for tracking an object high in reliability, in which the disadvantages of the prior art described above are obviated and an object can be accurately detected and tracked even in the case where another object passes the front of the target object.
In order to achieve this object, according to one aspect of the invention, there is provided an object tracking method used with an object tracking apparatus in which a predetermined number of template images are registered, an object in the imaging field is automatically detected by template matching based on the predetermined number of the template images and the detected object is automatically tracked, the method comprising the steps of registering the images of a detected object as template images, detecting the position of an image associated with the maximum degree of coincidence between an input image signal and any one of the predetermined number of registered template images, determining the detected image position by template matching as the position of the object tracked, and updating the template image based on the detected position, thereby tracking an object in the imaging field.
According to another aspect of the invention, there is provided an object tracking apparatus comprising an imaging unit for imaging the monitor range for detecting and tracking an object in the imaging field, an image input interface for converting the video signals acquired by the imaging unit sequentially into an image signal, and an image processor for processing the image signal converted by the image input interface, wherein the image processor conducts the template matching between a predetermined number of templates and the image signals sequentially input from the imaging unit, the template matching position is corrected by determining the template matching position associated with the maximum degree of coincidence obtained by template matching as a new template matching position, the image at the newly corrected matching position is updated as a template, and the direction of the target object is detected from the newly corrected matching position, thereby tracking an object intruding into the imaging field of the imaging unit.
Further, an object tracking apparatus according to an embodiment of the invention comprises a pan and tilt head for changing the direction of the view field of the imaging unit, and a pan and tilt head control interface connected to the image processor for supplying a control signal for controlling the pan and tilt head, wherein the direction of the view field of the imaging unit is adjusted through the pan and tilt head control interface toward the direction detected by the image processor thereby to track an object intruding into the view field of the imaging unit.